APG Unicode Parser

Parsers created with apg-js and apg-lite operate on arrays of positive integers—typically representing character codes. The apg-unicode variant extends this by supporting typed arrays, enabling more memory-efficient parsing workflows for modern JavaScript environments.

Note: apg-unicode does not natively parse Unicode. Instead, Unicode handling must be implemented via SABNF grammar and application logic. Typed arrays and conversion utilities simplify this process. See ./examples/unicode for an illustration of UTF-8 and UTF-16 parsing without prior transformation.

Key Features

Typed Array Support

apg-unicode accepts the following input types:

Array
Buffer
Uint8Array
Uint16Array
Uint32Array
String (converted internally to Uint32Array of code points)

Using typed arrays—especially Uint8Array—can reduce memory usage by up to 75% for large UTF-8 files.

Substring Parsing

Efficiently parse substrings within large strings without slicing or reallocating. Ideal for partial parsing scenarios. See ./examples/substrings for usage patterns.

Parser Generation

Like apg-lite, apg-unicode does not include a parser generator. To generate a grammar object, for example:

npm run apg -- -i ./examples/stats/sip.bnf -o ./examples/stats/sip

GitHub Usage

Clone the repo and run the user application and examples from the root directory:

git clone https://github.com/ldthomas/apg-unicode.git
cd apg-unicode

Include the modules in an application with:

import { Parser } from './src/parser.js';
import { Ast } from './src/ast.js';
import { Trace } from './src/tracer.js';
import { Stats } from './src/stats.js';
import { utilities } from './src/utilities.js';
import { identifiers } from './src/identifiers.js';

To run the examples use:

Command	Description
`node examples/ast/main`	Demonstrates Abstract Syntax Tree (AST) usage
`node examples/trace/main`	Traces the parser through the parse tree
`node examples/stats/main`	Collects and displays node hit statistics
`node examples/substrings/main`	Parses substrings within a full input string
`node examples/unicode/main`	Parses UTF-8 and UTF-16 directly without prior transformation to code points
display `examples/web/web.html` in any browser	Illustrates running a parser in a web page. Note that `web-app.js` is created with esbuild from `app.js`. Use the script `npm run bundle`.

npm Usage

Install the repo from the npm registry. In the application root directory:

npm install apg-unicode

To access the modules in the application:

import { Parser, Ast, Trace, Stats, utilities, identifiers } from 'apg-unicode';

Documentation

The documentation is in in the code in docco format. To generate it use:

npm run docco

The documentation will then be in at ./docs/index.html

Or view it here on the APG website.

License

apg-unicode is licensed under the permissive MIT license.